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Objectives 

Support  for  one  graduate  was  awarded  to  augment  the  numerical  and  computational 
work  performed  under  AFOSR  Grants  F49620-94-1-0313  and  F49620-97-1-0185  in 
the  Computational  Mathematics  Program.  In  particular,  we  developed  and  imple¬ 
mented  parallel  algorithms  for  computational  electromagnetics  with  applications  to 
turbulence  control,  and  new  algorithms  for  velocity-vorticity  formulations  appropri¬ 
ate  for  high  Reynolds  number  flows.  These  algorithms  are  based  on  the  multiscale 
spectral  discretizations  for  unstructured  meshes  we  developed  in  the  parent  grants. 
The  implementation  was  performed  on  the  IBM  SP2  and  SP3  at  Brown  University 
and  MHPCC  as  well  as  on  different  parallel  platforms,  including  Linux  clusters,  using 
the  MPI  communications  protocol. 

1  Research  Project  Description 

We  developed  a  new  hierarchical  spectral  basis  appropriate  for  hp-finite  element  for¬ 
mulations  on  unstructured  and  hybrid  grids  consisting  of  tetrahedral,  prismatic  and 
pyramidic  subdomains.  This  unique  basis  is  the  building  block  for  the  accurate  (spec¬ 
tral)  solution  of  partial  differential  equations  in  complex  geometry  computational  do¬ 
mains.  An  important  feature  of  this  work  is  that  high-order  accuracy  can  be  obtained 
on  standard  unstructured  and  hybrid  CFD  meshes.  Moreover,  unlike  low-order  dis¬ 
cretizations  which  may  require  Delaunay  triangulizations  for  convergence,  the  new 
hierarchical  basis  can  be  used  in  conjunction  with  highly  distorted  meshes  without 
accuracy  degradation. 

The  accuracy  of  the  new  method  was  tested  both  in  two-  and  three-dimensions 
for  benchmark  Navier-Stokes  and  MHD  equations.  For  smooth  solutions  we  obtained 
exponential  convergence  irrespective  of  the  grid  distortion.  Based  on  this  new  formu¬ 
lation  a  flow  solver  (NEKTAR)  was  developed  in  order  to  be  used  in  turbulence  and 
MHD  simulations.  There  are  three  important  new  features  of  this  code  that  make  it 
superior  to  our  previous  spectral  element  flow  solvers: 

•  First,  it  is  fully  adaptive  as  h-  and  p-type  refinement  can  be  readily  applied  using 
the  hierarchical  structure  of  the  basis.  For  example,  each  edge  or  face  or  the 
interior  in  each  tetrahedron  could  have  different  modes  according  to  resolution 
demands. 

•  Second,  it  is  more  efficient  as  more  effective  preconditioners  can  be  constructed 


2 


using  multi-resolution  type  algorithms.  These  discretizations  has  also  been 
employed  to  solve  the  magneto-hydrodynamics  (MHD)  equations  as  decsribed 
in  the  original  proposal. 

•  Third,  the  development  of  Discontinuous  Galerkin  projections.  This  is  a  break¬ 
through  in  variational  numerical  methods,  which  may  have  great  impact  in  many 
AFOSR  applications.  It  allows  general  high-order  differential  operators  to  be 
discretized  with  finite  jumps  ( L 2  continuity)  across  subdomains.  This,  in  turn, 
implies  that  “sliding”  domains  (of  great  interest  to  weapons  simulation  technol¬ 
ogy)  can  now  be  discretized  with  no  extra  computational  effort. 

The  specific  tasks  completed  are: 

•  NEKTAR-MHD  simulation  code:  For  fully  three-dimensional  geometries 
we  extended  our  base  code  NEKTAR,  that  allows  for  spectral  discretizations 
on  three-dimensional  unstructured  meshes  consisting  of  tetrahedra  or  other 
subdomains,  to  include  the  MHD  formulation  described  previously.  Valida¬ 
tion  includes  comparisons  with  exact  solutions  such  as  the  three-dimensional 
Hartmann-Hunt  solution  as  well  as  comparisons  with  numerical  solutions  from 
our  validated  code  PRISM. 

•  Electro-Magnetic  Flow  Control:  Here  we  employed  the  NEKTAR-MHD 
code  to  examine  different  combinations  of  electric  and  magnetic  fields  for  sup¬ 
pressing  near-wall  turbulence  fluctuations.  Experimental  evidence  suggests  that 
the  direction  of  the  resultant  Lorentz  force  is  very  important  with  the  largest 
drag  reduction  produced  with  a  force  normal  to  the  wall.  This  finding  was 
studied  in  detail  in  simulations  using  slightly  conducting  fluids  such  as  ionized 
gas  and  salt  water,  leading  eventually  to  a  publication  in  Science,  May  2000. 

•  Velocity- Vorticity  Penalty  Formulation: 

The  objective  of  this  task  was  to  develop  a  parallel  vorticity-velocity  spectral 
element  method  for  the  incompressible  Navier-Stokes  equations  to  study  turbu¬ 
lence  in  complex  geometries.  To  understand  why  the  vorticity-velocity  spectral 
element  method  is  chosen,  consider  the  problem  of  high  Reynolds  number  flow 
past  a  cylinder.  Vorticity  dynamics  plays  a  crucial  role  in  the  separation  of 
the  boundary  layer  and  the  three-dimensionality  in  the  wake.  The  boundary 
layer  forming  around  the  cylinder  is  proportional  to  Re~  2.  At  the  surface  of 
the  cylinder,  vorticity  is  created  in  the  boundary  layer  and  shed  into  the  wake. 
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Since  it  is  necessary  that  at  least  a  few  mesh  points  are  in  the  boundary  layer, 
the  condition  that  6Re  ~  0(1)  must  be  satisfied  where  6  is  mesh  width.  As 
a  result,  one  of  the  difficulties  of  simulating  high  Reynolds  number  flows  is 
the  number  of  mesh  points  is  large  and  hence  the  computational  effort  is  pro¬ 
hibitively  large.  Another  major  difficulty  is  knowing  the  amount  of  vorticity 
created  at  the  boundary. 

A  new  parallel  three-dimensional  vorticity-velocity  formulation  was  developed 
based  on  a  rigorous  equivalence  theorem.  The  vorticity-velocity  formulation 
is  based  on  a  semi-implicit  temporal  discretization  and  the  spectral  element 
spatial  discretization.  The  penalty  method  based  on  the  semi-discrete  equations 
is  the  key  to  imposing  high-order  accurate  vorticity  boundary  conditions.  A 
distinction  of  the  formulation  presented  here  is  that  the  vorticity  and  velocity 
fields  are  expanded  in  the  same  discrete  space  (polynomial  order),  while  other 
low  order  finite  element  vorticity-velocity  formulations  expand  the  vorticity  in 
a  subspace  of  lower  order  than  the  velocity.  The  penalty  vorticity  boundary 
conditions  are  imposed  by  augmenting  the  weak  Galerkin  formulation  of  the 
vorticity  transport  equation  at  the  boundary. 

•  Computational  Steering  Environment: 

We  developed  a  new  computing  approch  in  simulation,  namely  computational 
steering,  where  the  developed  hierarchical  spectral  methods  are  used  on  grids 
changing  continuou  sly  in  time  by  following  an  adaptive  procedure.  We  have 
developed  and  performed  such  computational  steering  procedures  for  parallel 
computers  in  conjunction  with  interactive  graphics  in  order  to  introduce  a  new 
mode  of  Direct  Numerical  Simulation,  i.e.  the  dynamic  DNS,  dDNS. 


Personnel 

•  Faculty:  G.E.  Karniadakis  (US),  Professor  of  Applied  Mathematics. 

•  PhD  Students:  J.  Trujillo  (US  citizen,  minority)  and  R.M.  Kirby  (US  citizen). 
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Interactions /Transitions 


The  two  students  funded  by  AASERT,  J.  Trujillo  and  R.  M.  Kirby,  attended  every 
year  the  APS  Fluid  Dynamics  meeting  and  presented  results  related  to  this  grant.  In 
addition,  the  PI  (Karniadakis)  was  invited  to  present  the  AFOSR-sponsored  research 
at: 

•  MIT  •  Cornell  Theory  Center  •  Boston  University  •  Virginia  Polytechic  Insti¬ 
tute  •  Hong  Kong  University  •  International  Conference  on  Parallel  Algorithms  and 
Applications  (Wuhan,  China)  •  Institute  for  Scientific  Computing,  Chinese  Academy 
of  Sciences  •  Institute  of  Mechanics,  Chinese  Academy  of  Sciences  •  University  of 
Tokyo  •  University  of  Notre  Dame  •  University  of  California  Santa  Barbara  •  Wright 
Patterson  Air  Force  Base  •  City  College/Levich  Institute  •  AFOSR  Boiling  AFB  DC 

•  Nuclear  Regulatory  Commission  •  National  Institute  of  Standards  and  Technology 

•  AIAA  Conference  New  Orleans  •  Cornell  Workshop  on  POD-Galerkin  Models  • 
University  of  Michigan  •  AFOSR/Princeton  •  DARPA/NUWC  Workshop  on  EMTC 

•  DOE/Oakridge  Workshop  on  DGM  •  NSF  Woskshop  •  AIAA  Fluid  Dynamics 
Conference  on  LES  •  SIAM  Annual  Meeting/Symposium  on  MHD  •  ICOSAHOM’98 
Symposium  •  Japanese  Society  of  Fluid  Mechanics  30th  Anniversary  Symposium  • 
University  of  Tokyo  •  Turkey  Workshop  on  Industrial  and  Environmental  Applica¬ 
tions  of  DNS/LES  •  Caltech  Symposium  on  Validation  and  Verification;  •  Argonne 
National  Labs  •  Boston  University  •  Los  Alamos  National  Laboratory  •  University 
of  Texas  A  &  M  •  Americal  Physical  Society  • 

The  code  NEKTAR  is  an  open  source  code,  and  it  has  been  disributed  to  more  than 
two  dozen  Universities  and  Laboratories.  Some  of  them  include  AFB  Research  Lab  at 
Kirkland,  Boeing,  MIT,  Berkeley,  Caltech,  Cornell  University,  Penn  State  University, 
University  of  Wisconsin,  Imperial  College,  North  Carolina  University,  Florida  State 
University,  OAK  Ridge  Labs,  Nielsen,  Inc.,  etc.  There  is  limited  documentation  of 
the  code,  which  has  made  this  distribution  somewhat  difficult. 
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